Statistical Significance

Purpose and Rationale

Understanding statistical significance is crucial because:

Decision Making
- Provides a framework for making data-driven decisions
- Helps distinguish between real effects and random variation
- Guides interpretation of research findings
Research Quality
- Helps assess the reliability of findings
- Provides a standardized way to evaluate results
- Enables comparison across different studies
Scientific Communication
- Provides a common language for discussing results
- Helps prevent overinterpretation of findings
- Guides proper reporting of research

Why We Need It
- Sample data always shows some variation
- Need to distinguish real patterns from random noise
- Provides objective criteria for decision making
How It Works
- Compares observed results to what's expected by chance
- Uses probability theory to quantify uncertainty
- Provides a framework for making decisions

Concept	Description	Importance
P-value	Probability of observed or more extreme results	Measures evidence strength
Significance Level ( $α$ )	Threshold for decision making	Controls Type I error
Effect Size	Magnitude of the observed effect	Measures practical importance

P-value Range	Traditional Interpretation	Better Practice
$p < 0.01$	Strong evidence	Report exact p-value
$0.01 \leq p < 0.05$	Moderate evidence	Consider practical significance
$0.05 \leq p < 0.10$	Weak evidence	Discuss uncertainty
$p \geq 0.10$	No evidence	Note limitations

Misconception	Reality	Explanation
Proves $H_{0}$ is false	Only provides evidence	Based on probability
Measures effect size	Different from significance	Can have small p-value with tiny effect
Guarantees importance	Only indicates reliability	Need to consider practical significance

Aspect	Description	Importance
Evidence Measure	Strength of evidence against H₀	Guides decisions
Reliability Indicator	Suggests result is not due to chance	Supports conclusions
Decision Tool	Helps choose between hypotheses	Guides actions

Before Analysis
- Set appropriate significance level
- Consider practical significance
- Plan adequate sample size
During Analysis
- Calculate exact p-values
- Consider effect sizes
- Check assumptions
After Analysis
- Report exact p-values
- Discuss practical significance
- Consider limitations

Pitfall	Problem	Solution
P-hacking	False positives	Pre-specify analyses
Multiple testing	Increased Type I error	Adjust significance level
Ignoring effect size	Missing practical importance	Always report effect size